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Abstract 

An image articulation manifold (lAM) is the collection of images formed when 
an object is articulated in front of a camera. lAMs arise in a variety of image 
processing and computer vision applications, where they provide a natural low- 
dimensional embedding of the collection of high-dimensional images. To date 
lAMs have been studied as embedded submanifolds of Euclidean spaces. Un- 
fortunately, their promise has not been realized in practice, because real world 
imagery typically contains sharp edges that render an lAM non-differentiable and 
hence non-isometric to the low-dimensional parameter space under the Euclidean 
metric. As a result, the standard tools from differential geometry, in particular 
using linear tangent spaces to transport along the lAM, have limited utility. In this 
paper, we explore a nonlinear transport operator for lAMs based on the optical 
flow between images and develop new analytical tools reminiscent of those from 
differential geometry using the idea of optical flow manifolds (OEMs). We de- 
fine a new metric for lAMs that satisfies certain local isometry conditions, and we 
show how to use this metric to develop a new tools such as flow fields on lAMs, 
parallel flow fields, parallel transport, as well as a intuitive notion of curvature. 
The space of optical flow fields along a path of constant curvature has a natural 
multi-scale structure via a monoid structure on the space of all flow fields along a 
path. We also develop lower bounds on approximation errors while approximating 
non-parallel flow fields by parallel flow fields. 
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1. Introduction 

1.1. Image articulation manifolds 

Many problems in image processing and computer vision involve image en- 
sembles that are generated by varying a small set of imaging parameters such as 
pose, lighting, view angle, etc. of a fixed three-dimensional (3D) scene. As the 
parameters vary, the images can be modeled as a lying on a (typically nonlinear) 
manifold called an image articulation manifold (I AM) [llElElB-J. Each point on 
an lAM is an image at a particular parameter value. Over the past decade, there 
has been significant work [S HI 17]| in learning and processing the underlying 
geometric structures associated with image ensembles. For instance, tasks such as 
recognition, classification, and image synthesis can be interpreted as navigation 
along a particular I AM. 

More specifically, we define an image articulation manifold (lAM) as the set 
of images M formed by the action of an imaging map i on a space of articulations 
6, i.e, M = {i0 = i{9) : 9 G B}. This imaging process can be decomposed 
into two steps: first the action of the articulation on a 3D object or scene and 
then the subsequent imaging of the articulated object/scene (see [8J for a detailed 
discussion of image formation)!^ 

1.2. lAM Challenges 

In spite of much progress, there are fundamental challenges to successfully 
applying manifold-processing tools to generic image data, in particular lAMs. 

First, it has been shown that lAMs containing images with sharp edges are 
non-dijferentiable |1|. Specifically, given parameters 6i and 62 with correspond- 
ing images Ii and I2, it follows that the distance ||/i — /2IIL2 between images 
Ii and I2 is a nonlinear function 77(||6^i — 6^2!!) that is asymptotically equivalent to 
(11^1 — ^2 11)^. Indeed, ^^||^^~^^J^|^ ^ c\\Oi — 02\\~^ , and this non-Lipschitz relation 



^Before we proceed further, it is worth discussing certain degeneracies in the imaging process 
that we wish to avoid in this paper for analytical reasons. In particular, we want to avoid cases 
where the set M is not expressive of the full range of articulations. As an example, consider a 
uniformly colored 3D sphere O undergoing rotation about a fixed axis. Here, the parameter space 
6 is the unit circle . Being uniformly colored, the sphere does not change appearance under 
rotation, and the I AM degenerates to a single point {lo}- Were the sphere richly textured, we 
would obtain new views of the sphere for each rotation so that the I AM is isomorphic io . For 
the remainder of the paper, we will consider lAMs without degeneracies by assuming that the 
imaging map i is a re-parametrization of B; i.e., we will assume that the lAM is homeomorphic to 
the corresponding parameter space. 
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Figure 1: Non-differentiability of lAMs renders (locally) linear transport inaccurate. Consider an 
image interpolation task. Given images Ii and I2, the line connecting them a{t) = t/i + (1 — t)l2 
is blurred and thus a poor approximation to the geodesic connecting Ii and I2 . 

indicates that the corresponding lAM is non-differentiable. Non-differentiabihty 
suggests that local linear approximations, such as those suggested by differential 
geometry, are invariably inaccurate on lAMs. To illustrate this, consider a stylized 
example of image interpolation (see Fig. [T]). Given images h.h ^ M, consider 
the affine path a{t) = tli + (1 - t)/2 with t G [0, 1] ; for a smooth manifold (> C^), 
this line would be a close approximation to the actual manifold, especially over 
small neighborhoods. However, non-differentiability implies that a first-order ap- 
proximation to the manifold is inaccurate even over a small neighborhood. This is 
illustrated in Fig.[T] To alleviate the non-differentiability problem, Wakin et al. ^ 
have proposed a multiscale smoothing procedure that regularizes each point of the 
lAM by a set of multiscale Gaussian smoothing filters that render the lAM smooth 
across the various scales. This smoothing procedure then enables the definition of 
linear tangent spaces on which one can perform standard linear methods of analy- 
sis. However, this is unsatisfactory, since Gaussian smoothing is inherently lossy 
— leading to loss of high-frequency information in the images. Further, defining 
tangent vectors as the limit of a multiscale procedure is inherently complex and 
moreover not possible for practical scenarios where we have only samples from 
the lAM. 

Second, conventional manifold models lack a meaningful metric between 
points on the lAM, especially when the sampling of the manifold is sparse. Con- 
sider the simple translation manifold Mt generated by imaging a black disk of 
radius R translating on an infinite white background (see Fig. [2]). Let Ii be the 
image of the disk with center ci and I2 the image of the disk with center C2. It 



3 




(c) Image I 3 (d) Image I4 



Figure 2: Euclidean distance between images on an I AM can be meaningless, (a-d) Images of a 
translating disk over a white background, (e) The distance between two images depends on the 
amount of overlap between the the disks. However, the overlap is zero when the distance between 
centers exceed 2R; in the figure above, d{Ii 5/3) = d{Ii ,14). Hence, the distance between two im- 
ages can be written as 7^(min(2i?, ||ci — C2 II2)) where r]{-) is a monotonically increasing function. 
Conventional manifold learning techniques, which rely heavily on meaningful local distances, do 
not work unless the sampling on the manifold is dense. 



then follows that 

\\h - h\\L^ = 7y(min(2i?, ||ci - C2I 



f^(||ci-C2||), IIC1-C2II <2R 
\r]{2R), IIC1-C2II >2R 



where 7y(-) is a nonlinear function. Thus, when the disk centers are separated by 
a distance greater than the diameter of the disk, the metric is completely unin- 
formative (see Fig. [2]). This suggests that, unless the sampling of images from 
the lAM is sufficiently dense, organization of the images using a construct such 
as a fc-nearest neighbor graph is meaningless, fc-nearest neighbor graphs are at 
the heart of traditional manifold learning techniques such as LLE [6J, ISOMAP 
PI, and diffusion maps [3]; as a consequence, such techniques are doomed to fail 
unless the sampling is sufficiently dense. 

These shortcomings are exacerbated for images with rich textures; this rules 
out a consistent analysis of lAMs based on classical differential geometry that 
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relies on the smoothness and metric properties of the manifold. A critical missing 
link in the analysis of lAMs is a systematic theory that successfully handles the 
above issues and permits new mathematical tools for lAMs, including analogues 
of traditional notions such as curvature, vector fields, parallel transport, etc. 
Clearly, once such analytic tools are available, we can greatly expand the scope of 
applications and pave the way for efficient algorithms specific to lAMs. 

1.3. Transport operators 

Recently, a new class of methods for handling image ensembles has been de- 
veloped based on the idea of transport operators (H [TOl [HI [121 IS. A transport 
operator on an lAM is a (typically nonlinear) map from the manifold into itself 
that enables one to move between different points on the manifold. Given im- 
ages /i(x) and /2(x), (where /(x) denotes the intensity at the spatial location 
X = (x, y) G [0, 1] X [0, 1]) a transport operator T is a mapping that acts in the 
following fashion: 

/2(x) = /i(x + r(x)). (1) 

Instead of relying on the linear tangent space that accounts only for infinitesi- 
mal transformations, nonlinear transport operators can be well-defined over larger 
regions on the I AM. 

For certain classes of articulations, the associated transport operators have al- 
gebraic structure in the form of a Lie group fTTI fTOl fT2l l4l|. In such instances, it 
is possible to explicitly compute transport operators that capture the curved geo- 
metric structure of an lAM. An example of this is the case of affine articulations, 
where the transport takes the form 

/(x)=/o(Ax + t). 

In this case, the transport operator r( ) is of the form r(x) = {A — ][)x + t; this 
can be modeled as the group of 2D affine transformations. The affine group has 
found extensive use in tracking [12J and registration [fT4|| . Miao and Rao [lOJ learn 
affine transport operators for image ensembles using a matrix exponential-based 
generative model and demonstrate improved performance over locally linear ap- 
proximations. Culpepper and Olshausen [4 J extend this framework using a more 
complex model on the transport operator in order to model paths on image mani- 
folds. 

Other common examples of articulations in computer vision are the projective 
group (used to model homographies and projective transformations) and diffeo- 
morphisms (used to model ID warping functions, density functions) fll5L How- 
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ever, while algebraic transport methods are mathematically elegant, they are appli- 
cable only to a very restrictive class of lAMs. Many lAMs of interest in computer 
vision and image processing applications, including lAMs corresponding to 3D 
pose articulations and non-rigid deformations, possess no explicit algebraic struc- 
ture. 

1.4. Optical flow-based transport 

In this paper, we study a specific class of transport operators that are generated 
by the optical flow between images (we introduced this notion empirically in [13J). 
Given two images Ii and I2, the optical flow between them is defined to be the 
tuple (v^, v^) G L2([0, If) X L2([0, 1]^) such that 

hix.y) = Ii{x + \^{x,y),y + \y{x,y)). (2) 

A common assumption in computing the optical flow between images is bright- 
ness constancy U6L where the spatial intensity is assumed to not change between 
Ii and I2. Since the pioneering work of Horn and Schunk [16], there has been 
significant progress towards the robust estimation of optical flow between image 
pairs (nlll. 

Optical flow is a natural and powerful transport operator to transform one im- 
age into another. In the context of image manifolds, the collection of all optical 
flow operators at a point on an lAM is a manifold of the same dimension as the 
lAM ||T3l|. In other words, at any reference image on an lAM, there is a corre- 
sponding manifold of optical flow operators that transports the reference image 
along the lAM. This new operator manifold, which we christen the optical flow 
manifold (OFM), can be used to obtain a canonical chart for the lAM [il3]l : this 
enables significantly improved navigation capabilities on the lAM than previous 
methods. In particular, OFM-based transport is well-defined even in instances 
when the transport operators cannot be modeled as a Lie group; an example of 
this is the pose manifold (the lAM associated with rigid body motion). 

To see the efficacy of optical flow-based transport, consider again the case of 
image interpolation, but now with optical flow as the transport operator (see Fig. 
|3]). Here, the path 7(t), t G [0, 1] generated on the lAM via optical flow is a better 
representative of a path on the lAM, i.e., j{t) ^ M for all t. Moreover, if the lAM 
is generated via Lie group actions, then this path coincides with the geodesic. 

As discussed earlier, lAMs composed of images with sharp edges and textures 
lack smoothness and hence do not support locally linear modeling. In contrast, 
for a large class of interesting articulations, including affine transformations and 
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Figure 3: Optical flow based transport on an lAM leads to accurate image interpolation. Consider 
again the interpolation task in Fig. [T] While locally linear transport on lAMs lead to inaccurate 
interpolation results, the path ^{t) generated using optical flow provides an accurate approximation 
to the true path between Ii and I2. Moreover, if the I AM is generated via Lie group actions, then 
the path generated coincides with the geodesic. 



3D pose, the corresponding OFMs are smooth (see [Appendix A| and lfT3]| ) and 



support local linear modeling. Moreover, we can define the distance between 
two optical flows to measure the amount of motion required to articulate from 
one image to another. These properties suggest that machine learning techniques 
(such as LLE, ISOMAP, etc.) should be able to extract a considerable amount 
of geometrical information about an image ensemble when applied to its OFMs. 
Figure |4] confirms this fact by showcasing the improvement in OFM vs. lAM- 
based dimensionality reduction for an object under 2D translation. 

The stability of OFM-based modeling enables us below to develop new dif- 
ferential geometric tools for signal processing such as the geometric mean and 
Karcher mean [19 ] of an image ensemble. The Karcher mean of a set of images is 
defined as the image on the lAM that is closest to the set in the sense of geodesic 
distance. Figure |5] shows the results of Karcher mean estimation using the OFM. 
The interested reader is referred to [13J for details of the Karcher mean estimation 
as well as more OFM-based tools and applications. 

7.5. Related work 

OFMs fall under the category of deformation modeling, which has been stud- 
ied in many different contexts, including active shape [20 J and active appearance 
models [21 J. However, there are significant differences between OFMs and tra- 
ditional deformation modeling approaches. Morphlets [22J, for examlpe, provide 
a mutliscale modeling of and interpolation across image deformations, but their 
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(b) Ground truth (c) lAM (d) OFM 



Figure 4: Comparison of manifold learning on a I AM vs. OFM (from |T3l). The I AM is generated 
by cropping patches of size 200 x 200 pixels at random from an image — thereby generating a 2D 
translation manifold, (a) Sample images from the I AM showing a few images at various transla- 
tions, (b-d) 2D embedding obtained using ISOMAP on (b) the ground truth, (c) the lAM, and (d) 
the OFM. The near perfect embedding obtained using OFM hints at the near-perfect isometry in 
the OFM distances. 

treatment is limited to image pairs. In contrast, OFMs apply to image ensembles 
consisting of a potentially large number of images. Beymer and Poggio ||23]| have 
argued for the use of motion-based representations for learning problems. How- 
ever, their goal is image synthesis, and therefore they offer no insights into the 
geometric nature of manifold- valued data. Jojic et al. [24J use a layered represen- 
tation to represent videos by separating the appearance of moving objects from 
their motion and then representing each using subspace and manifold models, 
respectively. This simple, yet powerful, representation can model and synthesize 
complex scenes using simple primitives, but it is not intended to go beyond simple 
manifolds such as those generated by translations and affine transformations. 

1.6. Specific contributions 

The main aim of this paper is to develop the mathematical foundation of op- 
tical flow-based transport operators for image manifolds, which were introduced 
empirically fV^. 

We first define a metric on an lAM using its corresponding OFMs. Each OFM 
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# # •# 



(a) Sampling of images from an I AM 



(b) Ground truth (c) lAM-based (d) OFM-based 

Karcher mean Karcher mean 

Figure 5: Karcher mean estimation for 20 images generated by the rotation of an object about a 
fixed axis ifTSl . The images are from the COIL dataset 1251 . (a) Sample images from the lAM 
showing a few images at various rotations. We estimate the Karcher mean using local linear trans- 
port on the I AM and the OFM. Shown above are (b) the ground truth Karcher mean of the images, 
(c) the Karcher mean estimated using local linear transport on the lAM, and (d) the Karcher mean 
estimated using local linear transport on the OFM. The accuracy of the estimate obtained from the 
OFM showcases the validity of local linear transport on the OFM. 

has a natural metric that is a locally isometric function of the corresponding pa- 
rameter values. We consider the induced metric on the I AM, which we dub the 
flow metric and show that the flow metric between two points of the lAM is a 
measure of the distance between the corresponding parameter values. Next, us- 
ing the flow metric, we develop analytic notions of curvature, optical flow fields, 
and parallel transport. We analyze in detail the case of optical flow fields defined 
along a fixed curve. In particular, we define a unique function associated to each 
such flow field, which we dub the motion function, using which we can define 
the notion of parallel flow fields. We answer the natural question of how one can 
optimally approximate a non-parallel field by a parallel field, and thereby induce 
uniform motion along the curve. We also construct a monoid structure on the set 
of all flow fields along a fixed curve. Under certain conditions on the curvature of 
a curve, we show that the space of parallel optical flow fields along the curve is a 
submonoid that comes with a convenient multi-scale structure. 

We envision that the theory developed in this paper will enable a large class 
of practical applications involving image manifolds especially under sparse sam- 
pling of images from the manifold. In this context, we believe that the long term 
impact of this paper is the first step towards a complete theory of manifolds for 
arbitrary classes of signals using transport operators along the lines of the classic 
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differential geometry for smooth manifolds. 
7.7. Organization 

The remainder of the paper is organized as follows. In Section[2} we introduce 
OFMs and, using a fixed metric on the OFMs, study the induced flow metric on 
the corresponding I AM. We compare dimensionality reduction techniques on the 
lAM vs. OFMs. Using the flow metric, we develop geometric tools on the lAM in 
Section|3]and highlight their application to parameter estimation. We also develop 
error bounds on approximating non-parallel flow fields by parallel flow fields and 
illustrate the idea with the example of video resampling. In Section |4] we develop 
the multiscale structure of parallel flow fields. We conclude in Section [5] with a 
brief discussion. 

2. OFMs and the Flow Metric 

In this section, we define and study the basic properties of OFMs correspond- 
ing to an I AM M. Much of the section is concerned with a formal introduction to 
OFMs, first defined in [13J, leading towards the development of our fundamental 
tool, the flow metric on an I AM. 

2.7. Optical flow manifolds 

The optical flow between two images on an lAM measures the apparent mo- 
tion between the two images and thus reflects the corresponding parameter change 
between the two images. For a fixed base image m G M, consider a neighbor- 
hood N{m) of m. If for G N{m) there exist flow vectors (v^;, v^) such that 
can be obtained from m using the flow vector, i.e., m\x, y) = m{x + v^^, y + v^), 
then we say that optical flow exists from m to m^ We denote this situation as 
(l)y^,yy{rn) = or simply (p{m) = m^ In practice, occlusion or boundary effects 
(i.e., veiling of certain portions of an image due to changes in the background 
or interferers in the scene) may lead to undefined estimates for the flow vectors. 
However, as described in ifTSll , one can mitigate these issues by incorporating 
additional consistency tests to ensure that only the meaningful flow vectors are 
retained. 

The set of all points m' ^ M for which optical flow from m ^ M iom' exists 
is a neighborhood of m, which we denote by 

Bjn = {m' eM \ m'{x, y) = m{x + v^, + v^)}. 

Using this neighborhood, we define the optical flow manifold Om at m as follows. 
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Definition 2.1. Let M be a K -dimensional I AM. Given m ^ M, the Optical Flow 
Manifold Om is defined as the set of optical fiows (j) between m and points in 

Om = {0 = (Vx, v^) : (j){m) G S^}. 

The neighborhood Bm is called the flow neighborhood around m. 

Om is nonlinear, i.e., an arbitrary OFM is not always a linear vector space. It 
is clear that the collection of neighborhoods {Bm : m G M} covers M. 

Proposition 2.2. Let M be a K -dimensional lAM. Then Om has the structure of 
a manifold homeomorphic to Bm- 

Proof We first show that Om is homeomorphic to Bm and puUback the smooth 
structure of Bm to induce a smooth structure on Om- Consider the map g : Bm 
Om that sends a point m' G Bm to G Om such that (f){m!) = m. g is clearly 
bijective and hence, we use g to endow Om with the quotient topology. Moreover, 
since g is injective, g is a homeomorphism since g is an open map: given an open 
set U C Bm we have U = g~^{g{U)) and by the quotient topology on Om, we 
have that V = g{U) is open in Om since g~^{y) = [/ is open in Bm- Since 
{Bm ' rn G M} is a chart for the manifold structure of M, there exists continuous 
maps {^m ' ^ ^ M} so that {{Bm, $m) ' ^ ^ M} is an atlas for M. Therefore, 
the composition $^ o g~'^ is a homeomorphism from Om to an open set W in R^. 
Since Om is now covered by the single chart {Om, $m 05^"^ }, we can puUback this 
smooth to Om via $^ o and endow Om with a smooth structure. Moreover, 
Bm is homeomorphic to Om by construction. □ 

We make a few preliminary observations. First, by the homeomorphic re- 
lationship between Om and Bm, we conclude that Om is also a X-dimensional 
manifold. Also, Omi is homeomorphic to Om2 for any mi,m2 G M. Moreover, 
the trivial element 00 G for each m e M that maps m to itself in Bm acts a 
natural "origin" in O^. 

We observe, again from the homeomorphic relationship between Bm and Om, 
that given any point mi G Bm there is a unique operator (/)i such that (m) = mi. 
A question to ask is whether there is any relationship between Om and the tangent 
space Tm at m. Indeed, for a smooth lAM, one can define tangent spaces Tm at 
m e M and the OFM Om at m is diffeomorphic to a neighborhood of G T^. To 
see this fact, recall that the exponential map defined on Tm is a diffeomorphism 
between a neighborhood [/q of G and a neighborhood of VmOfm. We also 
have a diffeomorphism $ from Om to by definition. Now, Bm is open and 
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there is an open ball Bq ^ Uq on which o exp is a diffeomorphism, being a 
composition of two diffeomorphisms. 

As a concrete example, consider again the translational manifold Mt from 



Section L2 Note that the parameter space in this case is B = M^. Since there is no 
occlusion between any two images on Mr, it follows that given any m, G Mr, 
there exists a G Om such that (/){m) = m^ Since this is true for any pair of 
images in M^, we conclude that Om Vm G Mt and hence, Mt 

Vm G Mt. More generally, we note that for an I AM M generated by Lie group 
actions without occlusion between images, the OFM Om at any point m e M can 
be identified with the parameter space B and neighborhood Bm at each point is 
the entire manifold M. In particular, one can recover the geodesic path between 
two any two points m, G M by using appropriate flow operators in Om to 
generate the geodesic path from m to m^ This shortest path corresponds to the 
geodesic in the parameter space between the parameter values corresponding to 
m, as well. We will return to this example in future sections and show how our 
more general formulation contains the algebraic methods such as ([HI [TOl [III 01 
as special cases. 

As in the case of the tangent bundle, we can construct an analogous bundle 
with the collection of Om since m varies through M. 

Definition 2.3. Let M be an lAM and Om the OFM atm e M. The flow bundle 

OM on M to be the disjoint union of the Om since m varies over M 

0M=i[ Om. 

Thus, an element of OM is a pair (m, cj)) with cj) G Om- Using this fact, we 
can induce a topology on the flow bundle. 

Proposition 2.4. Given a K -dimensional lAM M, the flow bundle OM is a 2K- 
dimensional manifold. 

Proof We first note that if (m, (/)) G OM then {m, (/>(m)} G {m} x Bm- Thus, 
given an atlas {{Ux, ^a)}aga we have that 

maps {m,(j)) into ipa{Ua) x H ^m) where Ua and Up are charts around 

m and (j){m) respectively. The (continuous) inverse of {xi, ...,Xk,yi, .--.yk} ^ 
i^a{Ua) X ippiUp fl Bm) is givcu by (m, 0) where ipaim) = {xi, Xk} and 
is the unique operator in Om such that j3{(j){m)) = {yi, ...,yk} so that OM is 
locally Euclidean. □ 
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Thus, we see that OFMs are manifolds consisting of flow operators that are 
defined pointwise on the corresponding lAM. The action of flow operators at a 
base point on the lAM results in motion along the lAM, as opposed to linear 
transport that results in motion off the manifold. 

The key property that makes the study of OFMs interesting is that, for interest- 
ing lAMs, the associated OFMs are smooth and exhibit nice distance properties 
[|l3jl. We summarize these in [Appendix A[ These two properties, namely smooth- 
ness and isometry, are in turn used to define a meaningful distance on the lAM. 
We discuss this next. 



2.2. Metric structure on an I AM via its OFMs 

Consider again the translational manifold M^, where the OFM Om at each 
point m G Mt can be identified with M^. Being isometric with M^, we can endow 
each Ojn with the Euclidean metric which we denote as (io(-, •)• Let 61,62 G 
be a pair of parameters such that mi = i{6i),m2 = ^(6^2); note that there exists 
a G Ojn^ such that (j){mi) = m2. It then follows that rfo(0o, 0) = C^H^i — ^2!! 
for some C > 0. As the results of lfT3]| indicate, the above discussion holds 
analogously for generic OFMs, i.e., each Om has an associated metric do{',') and 
this metric is locally isometric to a corresponding metric on the parameter space 
6. We indicate this as 

(io(0o,0) oc de{6i,62), 

where (po is the unique operator in Omi such that mi = 00 (mi). 

Our main focus in the remainder of this section is to define a corresponding 
metric for lAMs using the metric do{'r) on Om- The resulting metric on M 
inherits the property of being locally isometric to the changes in parameters. As 
a first step, we locally "push forward" the metric on Om onto Bm as follows. 
For points mi, m2 G M with m2 G Bm^, we have a unique operator cpi such that 
m2 = 01 (mi) so that we can define the distance dM{rni, 1712) as the corresponding 
distance between 00 and 0i 

dM{mi,m2) := rfo(0o,0i). 
Moreover, if mi = f{6i) and m2 = 1(62) for parameters 61,62 ^ then we have 

dM {mi, 1712) oc de{6i,62). 

However, this definition does not readily extend to the case where mi and m2 are 
not "optically related", i.e., m2 ^ Bmi- In this case, we first connect mi and m2 
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by a path c such that c(0) = mi, c(l) = m2. We then partition the domain of c by 
a partition P = {0 = to <^i < ••• <^n = l} such that the intermediate points 
along the path are optically related i.e., c(t^) G Bc(ti_i), where we assume that, for 
a fine enough partition, we can obtain such a nesting. We then define the distance 
along c to be 

n-1 

d{c,mi,m2) = sup V'(iM(c(ti),c(t^+i)), 
p 

where the supremum is over all partitions of the path.By taking the infimum over 
all possible paths, we obtain a metric on M, i.e., 

n-1 

dM{mi,m2) = inf V'(iM(c(ti),c(t^+i)). (3) 

c ^ — ' 

In essence, the metric dM{'r) is similar to the Riemannian distance: we first 
define distance over a fixed curve and then take the infimum over all possible 
paths (see Fig. [6]). 

Proposition 2.5. For an lAM M, the distance dM{'^ •) in ^ is a metric on M. 

Proof. Positivity of dM{'r) is clear, as is the fact that dM{rn,m) = 0. 
If rui ^ 7712, then for every path c between mi and m2, we have that 

n-1 

(iM(c(ti), c(t^+i)) 7^ and hence, dM{mi,m2) 0. Symmetry follows 

i=0 

from the fact that along c, c(t^) G Bc{t^_^) and hence, rfM(c(ti), c(t^+i)) = 
<^M(c(ti+i), c(t^)). For the triangle inequality, we note that given paths ci and 
C2 from mi to m2 and m2 to 7773 respectively, the path ci * C2 obtained by 
traversing ci and C2 in succession at twice the rate (i.e. ci * C2{t) = ci(2t) for 
^ t ^ I and ci * C2(t) C2(2t — 1) for | ^ t ^ 1) is a path from mi to 7713 

n—1 n—1 n—1 

and ^(iM(ci(ti),Ci(t^+l)) + ^ rfM(c2(ti), C2(ti+l)) ^ J]]rfM(ci * C2(ti),Ci * 
1=0 1=0 1=0 

^2(^2+1)). By taking the infimum over all such paths, we verify the triangle in- 
equality. □ 

Recall that the metric on Om satisfies rfo(0o, 0) oc rfe(^i, ^2) with (j) G Om 
and 61,62 the parameters corresponding to 777 and (j){m) respectively. From the 
above result, we see that, with the metric dM{',') on M.v^t have 

dM{mi,m2) oc rfe(<9i,6>2), 
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Figure 6: A pictorial representation of the OFMs and the flow metric for an I AM. Each OFM is 
mapped homeomorphicafly onto the flow neighborhood Bm of the corresponding base point. The 
optical flow distance between two points is computed by considering the infimum over all possible 
curves of the piecewise flow distance dM{c{ti)^ c(t^+i)). 

where mi and 1712 are points on M corresponding to the parameter values 9i and 62 
respectively under the assumption that the monotonicity of do c>ver different 
{Om} is universal, i.e., the distance between two points mi and 1712 does not 
change when we change our definitions of flow neighborhoods. We refer to this 
metric on M as iht flow metric. We note that the flow metric is dependent on the 
OFMs Ojn. We can now state the main result of this section. 

Theorem 2.6. Let M be an lAM and let dM{-^ •) be the associated flow metric. 
Then, rfM(^i, '^2) ^ ds{9i^ 62) as defined in ^ with mi and m2 points on M 
corresponding to the parameter values 61 and 62 respectively. 

The smoothness of the OFMs makes them amenable for use with conventional 
manifold processing tools. For instance, common dimensionality reduction meth- 
ods such as [[5l|6l|7j, etc. assume that the manifold is smooth and, hence, are tools 
that are more appropriate for use with the flow metric. As an example, consider 
Fig.|4| Here, we see that the pairwise flow metric between points varies smoothly, 
as opposed to the Euclidean metric on the lAM, where due to sharp edges, the 
matrix of pairwise distance has large off-diagonal entries. Moreover, the resid- 
ual error of non-linear dimensionality reduction using the flow metric decays very 
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rapidly. As a result, it is more tractable to analyze lAMs using the flow metric as 
opposed to the conventional Euclidean metric. 

As an example, consider the problem of estimating 9 e Q such that i{9) = m 
given a finite number of template points mi , • • • , G M with neighborhoods 
Bmi r ' ' Bjun that cover M. As a motivating special case, consider first the sit- 
uation when M = M^, the translational manifold with B = and a single 
template point m^ A similar problem has been dealt with in [lOl |4J where the 
authors estimate the Lie group generators corresponding to the lAM. Given the 
parameter value 9' = {c[, 4) of the base image m\ we can compute the the pa- 
rameter 9 corresponding to m as follows. First, we find the unique flow operator 
(/) = (01, G such that (p{m') = m. The parameter value 9 corresponding to 
m is then obtained as 6^ = {c[ + 01, C2 + 02). 

An entirely similar result holds for generic lAMs generated by Lie group ac- 
tions. From this simple example, we see that finding the optimal 9 is equivalent to 
finding the optimal flow operator that minimizes dM{(p{rn'),rn) with G O^/. 

We return to the general case with n template images and seek the op- 
timal flow operator G Orm that minimizes dM{(p{Tni),m), i = I,-- - ,n. 
Here, a single neighborhood does not cover the entire lAM, and, hence, to es- 
timate we first find the neighborhood B^h G • • • B^^} such that m = 
argmin^/^l^^ ... ,^^} (iM(^, ^0- search is then restricted to the neighbor- 
hood Brh. Within this neighborhood, we find the optimal G as above, i.e., 
= argmin^^^^ dM{^{rh),m). In essence, we first find the optimal template 
point and then search within the corresponding OFM for the optimal flow opera- 
tor. If a single template point generates the entire lAM, then this procedure clearly 
reduces to the Lie group case discussed earlier. We thus see that the generic OFM 
formulation includes the algebraic methods of [[ini[I3 as a special case. 

The remainder of the paper is devoted to developing geometric tools for lAMs 
that leverage the flow metric. Note that, unlike the tangent space, the OFM has no 
linear structure and, hence, we do not have at our immediate disposal tools such 
as parallel translation, covariant derivatives etc. We will construct analogous tools 
for our purposes via the flow metric and hence open up a vista for lAM analysis. 

3. Geometric Tools for lAMs via the Flow Metric 

In this section, we develop the basic tools needed to analyze the structure 
of lAMs using the flow metric. We will pay special attention to flow operators 
defined along curves on the corresponding lAMs. Keeping computations in mind. 
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these tools will open the door to a variety of applications as mentioned in the 
Introduction and beyond. 

3.1. Flow radius 

We first seek an appropriate measure of the size of an OFM and the corre- 
sponding flow neighborhood. In classical differential geometry, one measures the 
radius at a point in terms of the injectivity radius [I9j using the Riemannian metric. 
In a similar fashion, we will measure the radius of a point m in an lAM M in 
terms of the flow metric. 

3.1.1. Flow Radius atm^M 

Definition 3.1. Given an lAM M and m G M define the flow radius or simply the 
radius at m as 

Tm sup dM{rn, n). 

Note that we may regard as a function from M to i.e. r(m) = is a 
map from M into the non-negative reals. Moreover, it is continuous as a function 
of m. Consider the variations of as m varies. If is large, then one can find 
a suitable operator cj) G Om that transports m to a far away point, with distance 
measured using the flow metric. Conversely, a small indicates that m can only 
be transported within a small region, or said differently, m obstructs transport on 
the I AM. Moreover, rapid changes in the magnitude of within a small neigh- 
borhood indicate that the manifold is not well-behaved near m. In particular, this 
indicates that there are several points close to m that obstruct transport while there 
are also several points that allow flow over large distances on the I AM. 

3.1.2. Flow radius for Lie groups 

A class of lAMs for which is very well behaved, indeed for which is 
a constant, are those generated by a Lie group action. As a motivating example, 
consider again the translational manifold M^. We note that as Om = for each 
m G Mr, it follows that = oc. If the parameter space B is compact, for 
instance, ifQ = S^ and we consider affine rotations of a base image m generating 
the lAM M, then we again have that = M and is a constant < c < oo. 
For a generic lAM M generated by Lie group actions with = M, it follows 
that the flow radius is a constant whose exact value depends both on the object 
being imaged as well as the nature of the articulation. 
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3.1.3. Flow curvature 

The above discussion indicates that the reciprocal — is a measure of "curva- 
ture". When is large (or infinite), then the lAM is can be thought as being 
"optically flat" at m in the sense that there is no obstruction to transport on the 
manifold at m. In the other limiting case, i.e., if approaches zero, the only 
operator in Om is the trivial operator 00 and hence, M is has high "curvature" at 
m. We thus have the following definition. 

Definition 3.2. Given an lAM M and m e M define the flow curvature or simply 
the curvature at m as — . 

While the traditional notion of curvature is a point property, the flow curvature 
depends on both the base point as well as its neighborhood properties. In terms 
of flow curvature, we can now state that if M is generated by a Lie group then 
M has constant curvature. The class of lAMs with constant curvature will play a 
prominent role in our later analysis. 

3.2. Optical flow fields 

In this section, we focus on a construct motivated by differential geometry, 
namely the idea of a vector field on a manifold. Recall that a vector field is a 
section of the the tangent bundle, i.e., a vector field is a map a : M ^ TM such 
that TT o a = idM, where tt is the natural projection from the tangent bundle and 
idM is the identity map on M. In an analogous fashion, we define an optical flow 
field, or simply diflow field, as a section of the optical flow bundle of M. 

While vector fields are defined generically on manifolds, the special class of 
vector fields along curves is especially important in differential geometry. Vector 
fields along curves give rise to tools such as parallel translation, Jacobi fields, etc. 

In the case of lAMs, which lack analytic structure in general, we will define 
optical fields along curves on an lAM and recover similar geometric tools using 
the flow metric. As we assign operators to points on the curve, we intuitively 
would like the transport induced by the operator to remain along the curve so that 
the collection of flow operators induces motion along the curve. This special class 
of optical fields will be our main object of study for the rest of the paper. 

Definition 3.3. Let M be an lAM and let c be a smooth curve passing through 
mi, 1712 ^ M. Define 5^(^)1^^ = {c{t') ^ Oc{t) ' t' ^ t]- optical field 
from nil to m2 along c with mi,m2 ^ do is a map V : t OM such that 
V{t) := Vt G Oc(t) cind Vt{c{t)) G Bc{t) H ^ ^here by Vt{c{t)) we mean the point 
on the curve obtained by the action ofVt on c{t). 
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In other words, an optical field along c is an assignment of a flow operator Vt 
with Vt an element of the OFM at c{t) such that such that the action Vt on c{t) 
remains on the curve. Thus, the action of Vt on the base point c{t) at time t induces 
motion along the curve. When mi and 1712 are clear from the context, we simply 
refer to Vt as the optical field along c. 

In essence, the curve Vt traces a curve in OM as t varies with a consistent 
action on c(t). By Bc{t) f]cwe mean the intersection of the X-dimensional flow 
neighborhood Bc(t) with the one dimensional curve c starting at c{t) and hence, 
Bc{t) C\c is a one dimensional embedded curve in Bc{t) i-^-, Bc(t) f]c is a one 
dimensional "slice" of Bc{t)' We assume that Bc(t) H ^ connected. 

To measure the distance traveled by the action of Vt on c(t), we define a radius 
Tt restricted along the curve as opposed to the complete flow radius rc{t) at c{t) 

rt^ sup dc{c{t),n), 

^^^c(t) n c 

where dc( r) is the flow metric restricted to the curve c. Likewise, the curvature 
Kt along c is the ratio 

n 

Since Bc{t) Cl^ is connected and ID, the distance dc{c{t),m) between c{t) and 
^ ^ Bc{t) n ^ characterizes m in the following sense. Given any positive constant 
< r/ < rt there is a unique m G Bc{t) dc{c{t),m) = r/. Thus, by 

specifying the distance along the curve c, we effectively characterize the curve. 
Lifting this observation into Oc{t) ^ we have the following result. 

Theorem 3.4. Let M be an lAM, let c be a curve passing through mi, m2 G M, 
and let Vt be an optical field from mi to 7712 along c. Then, Vt is completely 
characterized by the function hv{t) rfc(Vt(c(t)), c{t)) in the sense that for any 
non-negative function h(t) bounded pointwise by Vt, there exists a unique optical 
optical field Vt such that hy{t) — h(t). 

Proof That < hv{t) < Vt is clear from the definitions above. Let h{t) be any 
non-negative function bounded by rt. Then, for fixed t = to we have that < 
h{to) < TtQ. By the remark made previously, we have a unique mt^ G Bc(to) f] ^ 
with dc{c(to),mtQ) = h(to). Now, as mt^ G B^^to) fl particular, mt^ G ^^(^0) 
and hence there exists a unique (/)tQ G Oc{to) such that 

0to(c(io)) ^rnto' 
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As to was arbitrary, as t varies, we can define Vt = (/)t' Moreover, 



hv{t) = d,{Vt{c{t)),a 



it)) = dMt{c{t)),c{t)) = d,{mt,c{t)) = h{t) 



so that h{t) characterizes Vt. 



Recall that we view an optical flow field Vt along c as a curve in OM, Theorem 
10 states that this curve is characterized by the function hv{t) in the sense that for 
a given Vt along the (fixed) curve c, the function hv{t) contains all the information 
about the motion induced by Vt on the curve i.e., hv{t) measures the distance to 
which Vt transports c{t) in Bc(t) f] ^- Since this function is of prime importance, 
we make the following definition. 

Definition 3.5. Given an optical field Vt along a curve c, define its motion function 
to be 



Note that by continuity of the metric dc{-,-) and c(t), the function hv{t) is also 
continuous. 

3.3. Parallel flow fields 

There is a very natural geometric interpretation of the motion function hy of 
an optical flow field Vt along a curve c. Namely, it is a measure of the distance 
traveled along the curve at time t by c{t) when acted upon by Vt. Thus, it is natural 
to think of instantaneous changes in hy in t as a measure of the velocity of the 
motion induced by Vt on c{t). In classical geometry, the class of constant velocity 
curves is especially important; they correspond to uniform motions. Similarly, we 
will be interested in the class of constant motion functions hy. These correspond 
to optical fields Vt along c that induce uniform motion along c, where by uniform 
motion we mean the distance traveled along c is constant for all time. 

Another key link with classical differential geometry is the notion of parallel 
transport (or parallel translation) of tangent vectors JSl. Parallel transport is a key 
tool for "moving" tangent vectors from different tangent spaces while preserving 
direction/orientation along the curve. Parallel transport utilizes the inherent linear 
structure of the tangent space to define a linear map between tangent on points 
along the curve. We aim to develop a similar analytic tool for lAMs. However, 
an immediate stumbling block is the clear lack of linear structure in the OFM. 
We therefore take a different approach to defining parallel translation in the OFM 
case using motion functions. 



hy{t)=d,{Vt{c{t)),0 



it))- 
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To motivate our definition, we recall that a vector field along a curve c on a 
manifold is parallel if its covariant derivative along c vanishes. The covariant 
derivative is in essence a way of differentiating the vector field along c. The rela- 
tion between parallel translation and parallel vector fields is that, given a tangent 
vector V in the tangent space of a point c{t) on the curve c, it is possible to extend 
V along c by parallel translation to yield a parallel vector field along c. In the OFM 
case, we have the motion function of an optical field along c at our disposal and 
we use it to characterized parallelism of the field along the curve. 

Definition 3.6. An optical field Vt along a curve c, is parallel if the derivative of 
its motion function with respect to t is zero, or equivalently, if the motion function 
is constant along c, i.e., hy{t) is a constant. 

In all that follows, we will denote by ri(c, 7711,7712) the space of all optical 
fields through 77ii and 7712 along a curve c passing through 77ii, 7712 G M. When 
77ii, 77i2 are understood from context, we simply refer to this space as Q.{c). The 
subclass of parallel fields along c will be denoted by uj{c). 

A few facts are immediate from the above definition. First, since hv{t) < Vt 
for all t, it is clear that if Vt is to be parallel along c, then hv{t) is a constant hy 
independent of t and hy < inf^ Vf. Therefore, in the future, we will suppress the 
argument t in the motion function hv{t) of a parallel flow field. Second, since Vt is 
characterized by hy, we see that for any constant 6 such that < (5 < inf^ r^, there 
is a parallel optical field along c{t) such that hy = 5. Such a parallel optical field 
can be obtained, for example, by choosing for each t a flow operator (/)t G Oc{t) 
with dc{(pt{c{t)) , c{t)) = 5. The existence of such an optical field (j)t is guaranteed 
by the above theorem. Given (j) G Ocq we aim to extend (p throughout the curve 
to obtain a flow field Vt such that Vt is parallel along c with hy = (ic(0(co), (co))- 
The following result allows for such parallel translation of flow operators along a 
curve. 

Proposition 3.7. Let M be an lAM, and let cbe a curve through 77ii, 7712 G M. 
Let e — inft rt. Then, given any (/) G Oc{to) ^ith 5 — dc{(j){c{tQ)) ^ <^(^o)) < ^ there 
exists a unique parallel optical field along c{t) with hy{t) — 5. 

Proof. As 5 < e, in particular 5 < rt. Thus, invoking the previous theorem for the 
special case of the constant function h{t) = 5, we have the existence of a unique 
optical field Vt along c{t) such that hy{t) = 6. Moreover, since hy{t) is constant, 
Vt is parallel. □ 
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In contrast with the classical case, parallel transport along a curve is dependent 
on the nature of the flow operator, i.e., an arbitrary flow operator cj) G Ocq cannot 
be parallel translated along c unless dc{(j){co) , cq) < inf^ rt. This constraint is 
related to the nature of the curve; parallel transport along a curve that contains 
points with high curvature Kf is limited to those flow operators that induce smaller 
motion along the curve. Moreover, the possibility of parallel transport of G Oc(t) 
has a global dependence, i.e., it depends on the curvature of the entire curve, not 
only the curvature Kf at the point c{t). 

Thus, we are naturally led to study those curves for which parallel translation 
of an operator at a single point c{to) ensures the existence of parallel translation 
of operators at any other point on the curve. Clearly, the necessary condition is 
the invariance of Kf with t, and therefore we are led to consider curves for which 
the curvature Kf is independent of t, i.e., a constant. This special class of curves 
has a very rich structure that we explore in the following sections. 

3.4. Approximation of arbitrary flow fields by parallel fiow fields 

In this section, we consider the problem of approximating an arbitrary Vt G 
n{c) by elements in caj{c). Consider an optical field Vt G ^{c) (not necessarily 
parallel) along a curve c of constant flow curvature. A natural question to ask 
is how far away Vt is from being parallel. One way to do this is to seek the 
"best" approximation of by a parallel field Wt G uj{c) along c. To quantify the 
approximation, we consider the following cost 



In words, e{t) is a measure of how the action of Vt on c{t) differs from the action 
of Wt on c{t). Note that e{t) is bounded above by hv{t) + hw since e{t) = 



4(V^(c(t)), Wt{c{t))) < d,{Vt{c{t)),c{t)) + d,{c{t), Wt{c{t))) = hy{t) + V. 



While e{t) is a pointwise error, we will need to consider the total error over 
the entire curve. To this end, a natural choice of error metric is 



where the domain of c is the interval (a, h). Thus, our goal is to find a parallel 
optical field Wt G uj{c) that minimizes E{y, W), i.e.. 



e{t) = d,{Vt{c{t)),Wt{c{t))). 



E(V, W) 




W ^SiTgmmE{V,W). 



w 
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In general, a minimizer may not exist, or it may not be unique if one exists. How- 
ever, the greatest issue is the strong dependence of the error e{t) on the flow 
metric, which prevents a generic solution to the minimization problem since we 
cannot infer the convexity of the problem as stated. We can, however, obtain a 
universal lower bound on the error independent of the flow metric as follows 

\hv{t) - hw\ = Kmc{t)),c{t)) - 4(^(c(t)),c(t))| < e(t). 
Therefore, we seek a minimizer of / \hv{t) — hw\dt. Note that since Wt is 

J a 

parallel, hw is a constant, say h > 0. Moreover, since c is a path of constant 
flow curvature, is a constant r > 0. Since h characterizes Wt, a minimizer /i* 
yields a lower bound for E{V, W). Our goal then, is to find an optimal constant 

h* that minimizes E{h) = / \hv{t) — h\dt. We claim that a solution h* is a 

J a 

certain "median" of hy{t). We first define = {t ^ hyit) > k} and 

= {t G (a, b) : hv{t) < k} for some k. We claim that an optimal constant h 
is such that A(A^) = A(S^), where X{S) denotes the measure of a set S. 

Theorem 3.8. Let h be the constant such that A(A^) = X{B^). Then, h minimizes 

m- 

Proof. Note that the function E{h) is convex in h with h G (a, b) and (a, b) a 
convex interval. Therefore, we are guaranteed a minimizer /i*. Now, without loss 
of generality, we assume that /i* < h. We evaluate the cost over the two regions 
\ and 5^ 



'E{h*) = ! \hv{t) - h*\dt = / \hv{t) - h*\dt + I \hv{t) - h* 

Ja J At Jbt 



\dt. 



Define Bi = {t : h* < hv{t) < h} and B2 = {t : hv{t) < h*} and note that 
\{Aj^) — X{Bi) — \{B2) — 0. We can now express E{h*) in terms of E{h) as 

E{}?) = EZ^ih) + ih- h*)XiA^) + E^^(h) -{h- h*)\{B2) - a, 

where Ex{h) denotes the cost function restricted to the subset X C (a, h) and a 
is a positive constant that measures the difference of h — h* on the set Bi . The 
maximum value of ais (h — h*)X{Bi) and hence 

E(J?) <EX^(h) + (h- h*){X{A-J - X{B,) - X{B2)) + E^^(h). 
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Since A(A^) - X{Bi) - \{B2) = 0, we conclude that 



and hence /i* = h. The case of /i* > /i follows from symmetry. 



□ 



We illustrate the approximation of non-parallel flow fields by parallel flow 
fields with the example of video resampling where we can consider a video I to be 
a curve on an lAM i.e., a video X = {/^ , < t < T} with T > 0. This application 
is related to the problem of dynamic time warping (DTW) (261 EH [2H]|, where 
one is interested in measuring the similarity between two sequences that vary in 
time or speed. As we shall see, this can be used for matching or aligning video 
sequences with a warped time axis Il27ll281. 

Consider the lAM generated by imaging a black disk on an infinite white 
background starting with an initial velocity and accelerating with constant ac- 
celeration a along a fixed direction. For instance, the disk can be thought of as 
undergoing freefall off an infinitely high cliff. The lAM is a ID curve c and 
homeomorphic to IR+, the non-negative reals. Note that the curvature is every- 
where zero since = oc. Given an arbitrary flow field Vt along c, our goal is 
to analytically construct a parallel field Vt such that Vt is the unique parallel flow 

field that minimizes E{h). We first consider the video obtained by the action of Vt 
i.e. I = {Vt{c{t))}. Since Vt is not parallel, the video will show the disk moving 
with non-uniform motion. Our goal is to make the video uniform, i.e., generate a 
new video I from Vt that shows the disk moving with uniform motion. 

From the physics of the problem, it is clear that dc{c{t), c{t + 5t)) = K{vtSt + 
^aS^) for some positive constant K and any time increment 5t with Vt being the 
velocity at time t. Now, since Vt is to be a parallel flow field, we have that hy is 
a constant denoted by h. Thus, dc{Vt{c{t)), c{t)) = dc{c{t),c{t + 5t)) = hso that 
K{vtSt + laS^) = h. Rearranging this equation, we arrive at 



Solving for St, we obtain two real roots S]''^ = ^ ± J + which the 



(physically meaningful) positive root = + y + ^ > Ois retained. 
Thus, by defining Vt such that Vt{c{t)) = c{t + S}), we see that Vt is the unique 

parallel flow field that minimizes E{h). The new video I = {Vt{c{t))} will 
thus show the disk moving with constant velocity. With this, we have effectively 
linearized the motion and made it independent of the acceleration of the disk. 




2vt^ 2h 
a ^ aK 



= 0. 
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4. Multiscale Structure of Parallel Flow Fields 

As indicated in the Section [3} the set of parallel optical fields is a very special 
subset of the set of all optical fields along a fixed curve c. In this section, we 
will construct a monoid structure (i.e., a set with an associative operation and 
identity) on the set of all optical fields along a fixed curve c and show that the 
class of parallel fields forms a submonoid of this set under some conditions on 
the curvature along the curve. Moreover, the monoid operation yields a multiscale 
structure on the set of parallel optical fields. 

4.1. Monoid structure on Vt{c) 

As noted previously in Section [2} a clear disadvantage in dealing with the 
space of all optical fields along a curve is the lack of a linear, or more generally, 
any algebraic structure. In order to remedy this situation, we will define a binary 
operation on the set of optical fields that yields a monoid structure. We first fix a 
curve c passing through mi, m2 G M. 

Recall that a generic Vt G Q.{c) is characterized by its motion function hv{t). 
Thus, operations defined on motion functions hv{t) translate to operations on 
Vt e^{c). With this in mind, we define for Vt.Wt e ^{c) the sum Vt + Wt to be 
the unique optical field with motion function 

hv+w{t) = mm{hv{t) + hw{t),rt). 

Since hyj^w{t) < for all t, we see that hv+w{t) corresponds to a unique flow 
field that we define to be Vt + Wt. This operation is clearly commutative. Note 
also that the trivial (parallel) field Zt defined to be the field that acts trivially on 
c(t); i.e., 

Zt{c{t)) = c{t) 

is characterized by the motion function hz{t) = since hz{t) = 
dc{Zt{c{t),c{t)) = dc{c{t) , c{t)) = 0. Moreover, for any Vt G f^(c), we have 
that 

Zt + Vt = Vt. 

We see that Zt acts as the identity element in Vt{c). In addition to this, there is also 
the unique optical field Ut characterized by 

hu{t) = n 

that satisfies 

Vt + Ut = Ut 



25 



for all 14 G fi(c) and hence acts as the "absorbing" element of Q(c). 

Clearly, we do not have "inverses" with respect to "+" in the sense that, given 
a generic Vt G f^(c), there does not exist a Wt such that Vt + Wt = Zf. However, 
we do have "conjugates" with respect to "+" in the following sense. Given any 
Vt G fi(c), there is a unique 1^* G fl{c) such that 

Vt + v; = Ut. 

V^* is defined by its motion function 

hv-{t) =rt- hv{t). 
We refer to V^ as the conjugate of Vt. Moreover, 

{v:r = V,. 

Finally, we note that Ut and Zt are conjugates. 

Proposition 4.1. Vt[c) is a monoid under the operation "+ defined in (22). 

Proof. We only verify associativity, since Zt provides the identity. Given 
Vt,Xt,Yt G n{c), we consider the sums {Vt + Xt) + Yt and Vt + \Xt + 1^). 
If hy{t\ hx{t), hrit) are such that hv{t) + hx{t) + hyit) < u then both sides 
are clearly equal. If on the other hand, hv{t) + hx{t) + /ly (t) > Vt then we must 
have that either the sum /ly (t ) + hx {t) + hY{t) taken two factors at a time exceeds 
Tt or that the combined sum of all three factors exceeds with the sum of no two 
factors exceeding r^. In the first case, we assume that hv{t) + hxif) > rt, which 
implies the sum 

(Vt + Xt)+Yt^Ut 

with {Xt + Yt)y^Ut, i.e., 

hx{t) + hY{t) <rt. 
However, Vt + {Xt + 1^) is characterized by 

mm{hy{t) + mm{hx{t) + hY{t),rt),rt) = min(/iy(t) + hx{t) + hY{t),rt) = n 
which shows that 

Vt + {Xt + Yt) = Ut 
as well. The other cases follow by similar arguments. □ 
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4.2. Parallel fields along curves of constant curvature 

The addition operation defined above restricts to the set uj{c) of parallel fields 
along c{t). However, for generic c(t), the sum Vt + Wt of two parallel fields 
Vt.Wt G uj{c) may result in a possibly non-parallel field. For instance, consider 
hv{t),hw{t) such that /ly(to) + W(^o) < but hvih) + hwih) > for 
some t^.ti. Clearly, the sum is not parallel since in the first case, hyj^wito) = 
hv{to) + hw{to) < Uo while hv-^w{ti) = so that hv+w{t) is not constant. 

However, that if rt is constant along c, i.e., the flow radius is constant along 
the path, then the above situation is vacuous. Since Kt = j-^ is the curvature along 
the curve c, we are essentially requiring the curvature to be constant along c. We 
formally record the above observation. 

Proposition 4.2. Let cbe a curve with constant curvature Kt. Then, the operation 
restricted to uj{c) is well defined and uj{c) is a submonoid ofQ{c). 

Proofi We need only verify closure of in uj{c) since Zt G uj{c). If Vt G uj{c), 
then hy{t) is a constant and we will therefore suppress the argument t in hy{t). 
Note that since c is of constant curvature, rt is a constant r > 0. Thus, if Vt.Wt G 
a;(c), then hyj^w — min(/iy + hw-^ t). If hy + hw < t then as hy and hw are 
both constant so is their sum hy + hw and 

hyj^W — hy + hw ' 

If /ly + /ivF ^ ^5 then 

^y+w^ = min(/iy + /iv^, r) = r. 
In either case, /ly+vF is constant and hence 

Vt + Wt^uj{c). 

□ 

Thus, curves c that have constant curvature at all points are very special; not 
only is parallel transport determined by a single point on the curve, but uj{c) is 
also a submonoid of Q.{c). 
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4.3. Multiscale structure ofuj{c) 

In this section, we look for finite submonoids of uj{c). We would like the 
submonoids to be canonically defined, by which we mean that they depend only 
on the geometry of the constant curvature curve c. In particular, we construct for 
each positive integer k, a collection of finite submonoids Vn,k of uj{c) with n a 
non-negative integer and c a constant curvature path. Moreover, we will see that 
this collection is naturally nested, i.e.. 

In essence, this provides a multi-scale view of uj{c). 

Since uj{c) consists of parallel optical fields, we will simply denote the motion 
function of G uj{c) as hy. Moreover, we will suppress the subscript t 

when referring to elements ofVt G a;(c). Also, for V G u;(c), we mean by ^ the 
element in uj{c) with motion function being the constant Recall also that uj{c) 
possesses two canonical elements Z,U G a;(c) n n{c) that act as the trivial and 
absorbing elements of uj{c) respectively. 

We begin now with the construction of Vn^k- Fix a positive integer k and set 

Vo,k-{Z,U}. 

Next, we inductively set 
Now, it is clear that 

Vo,fc c Vi^k c .... 

Moreover, by the constant curvature condition, the sum of any two elements in 
Vn,k remains in Vn,k while associativity is obtained from the corresponding prop- 
erty in uj{c). Finally, since Z G Vn,k for all n, fc, we conclude that each Vn,k is a 
finite submonoid of uj{c). 

We have thus obtained a sequence of finite submonoids of uj{c). Moreover, 
with increasing n, it is clear that an arbitrary V ^ uj{c) can be uniformly approx- 
imated by y G Vn,k in the sense that \hy — hy \ can be made arbitrarily small by 
choosing larger n. In other words, the sequence Vn,k is "dense" in uj{c). Finally, 
since the basic generators of Vn,k are Z,U G a;(c) n fi(c), this construction is 
canonical in the sense that it depends only on the set of {Z, C/}, which are in turn 
are characterized by the global geometry of the curve c. 
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In Section [3| we saw that with curvature conditions on a curve c, it is possible 
to approximate an arbitrary flow field by a parallel one. The results presented 
above show that one can further approximate a parallel flow field with a finite 
collection of "template" elements from u){c). This is very much similar to the 
multiscale representation that wavelets provide for natural images. Indeed, the 
multiscale structure inherent to monoids paves the way for "lossy compression" 
of arbitrary flow fields by storing only the relevant scales Vn^k- In addition to 
compression, the multiscale structure can potentially enable fast computations on 
the flow fields. Operations on the flow fields can equivalently be mapped to those 
on the Vn^k without loss in accuracy while gaining significantly in the number of 
computations required. 

5. Discussion 

In this paper, we have developed the mathematical foundations of optical flow- 
based transport operators for image manifolds, which were introduced empirically 
ifTSll . Our main theoretical contribution was the development of the flow metric 
for using the ambient metric on the OFMs. Using the flow metric, we derived dif- 
ferential geometric analogues of tangent bundles, vector fields, parallel transport, 
curvature etc. When the lAM is generated by Lie group parameters, we showed 
that the OFM framework includes previous algebraic methods as a special case. 
Moreover, since the flow neighborhood at each point is the entire lAM, we can 
obtain geodesies between any two images using flow operators. While this paper 
has focused on explaining and extending the results obtained in [13J, we envision 
that the theory could make an on a large class of applications involving image 
manifolds. 

A clear assumption in our analysis has been that optical flow is the transport 
operator of choice for lAMs. While this is true for a majority of lAMs gen- 
erated by motion-induced parameter changes such as translations, rotations and 
unstructured plastic deformations, there are classes of lAMs for which the choice 
of transport operator is not immediately clear. For instance, for illumination man- 
ifolds obtained by variations in the illumination of an object, optical flow may not 
be the transport operator of choice, since such manifolds do not in general obey 
the brightness constancy requirement needed in optical flow computations. More- 
over, in cases where there is significant self-occlusion during the imaging process, 
optical flow may not be a practical transport operator. However, by regularizing 
the optical flow computation to handle occlusions by removing flow operators 
that lead to undefined motion between pixels, one can partially circumvent this 
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issue, and the methods of this paper can be profitably apphed. However, these 
are aspects of a more computational nature, and we reserve them for an alternate 
forum. 

A number of avenues for future work exist. First, our work hints that it 
should be possible to develop additional geometric tools such as affine connec- 
tions, holonomy, etc. using flow operators. These will complete the lAM anal- 
ysis toolbox. Second, although our development has been specific to the case 
of lAMs with optical flow, the basic model is extensible to a wide variety of 
signal ensembles with appropriately defined transport operators. For instance, 
a manifold model for speech signals has been proposed in ll29l where appropriate 
transport operators and the analog of OFMs may involve a frequency domain ap- 
proach. Once the "right" transport operator has been identified for the application 
in hand, one can conceivably define and study metrics similar to our flow metric 
and thereby develop analytic tools for further analysis. 
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Appendix A. Smoothness and Distance Properties of OFMs 

In this appendix, we establish the smoothness and isometry properties of 
OFMs associated with two interesting classes of lAMs: the affine articulation 
manifold and the pose manifold. 

Appendix A.l. Affine articulation manifold 

Affine articulations are parameterized by a 6-dimensional articulation space 
6 = M^; each articulation 9 can be written as 6> = (^,t), with A G M^""^ and 
t G M^. Any image belonging to the lAM can thus be written as 

/l(x)=/ref((A + I)x + t), (A.l) 
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where x = (x, y) is defined over the domain X = [0, 1] x [0, 1] and where /ref is a 
reference imagej^ The optical flow field fe associated with the transport operator 
can be written as fe{^) = /A,t(x) = Ak + t. Now, recall that the OFM at /ref is 
defined as 

The linear dependence of the optical flow field on both A and t implies that the 
OFM, which is the collection of optical flows at /ref, is infinitely smooth. Finally, 
noting that the OFM is independent of the reference image, we can establish the 
following result. 

Lemma 1. For affine articulations, the OFM at any reference image is infinitely 
smooth. 

Next, we consider distances between optical flows and establish that the OFM 
is isometric. 

Lemma 2. For affine articulations, the OFM at any reference image is globally 
isometric. 

Proof. Given two articulations 9i = (Ai,ti) and 62 = (^2,t2), the Euclidean 
distance between them is given by 



id{h,Jejf = [ ||/,,(x)-/,,(x)||^dx 

11(^1 -A2)x+(ti-t2) 11^ rfx 
(^1-^2^2(^1-^2), 



[x'^ 1]^ dx. 



where 

For A' = [0, 1] X [0, 1], it is easily shown that S is full-rank and positive definite. 
Hence, we have 

d{fei,fe2) = 11^1 - ^2||e, 
where || • ||e is the Mahalanobis (or weighted Euclidean) distance defined using the 
matrix S. This implies that the OFM is Euclidean and, hence, globally isometric. 

□ 



^Boundary-related issues are always an concern when we define images over a finite domain. 
Here, we circumvent this by assuming that the regions of interest are surrounded by a field of 



zeros. This allows us to assign undefined values to zero and satisfy ( A.l 
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Appendix A.2. Pose manifold 

The pose manifold is the lAM corresponding to the motion of a camera observ- 
ing a static scene. It is well-known that the articulation space is 6-dimensional, 
with 3 degrees of rotation and 3 degrees of translation of the camera, i.e., 
6 = 5^0 (3) X M^. We assume that the optical flow is the (unique) 2-dimensional 
projection of the motion flow of the scene induced due to the motion of the cam- 
era. 

Without loss of generality, we assume the reference articulation ^ref = 
(i?o, to) = (I, 0) and that the camera's internal calibration is known and accounted 
for [l30l . At the reference articulation 9^.^^, let the depth at a pixel x be given by 
Aref (x). Under an articulation 9 = {Rq, te), the optical flow observed at the pixel 
X is given by 



/(^)(X) = P A,ef(x)i?, 



X 

1 



+ t - P A,ef (X) 



X 

1 



(A.2) 



where P(-) is a projection operator such that y, z) = {x/z, y/z). Note that 
P is a well-defined and infinitely smooth function provided z ^ Q. 

The OFM O is the image of the map f{9) = {/(6^)(x) : x G A'}, where X 
is the image plane. We now establish the smoothness of the OFM. 

Lemma 3. Consider the OFM corresponding to a pose manifold of a scene with 
depth map (at the reference articulation ^ref J strictly bounded away from zero, i.e, 
V X, Aref(x) > A > 0. Then there exists a neighborhood of 9^^^ where f{9) = 
{/(0)(x); X G A'} is an infinitely smooth map. 

Proof. The smoothness of f{9) follows from the smoothness of f{9){^), which 
in turn follows from the smoothness of the projection operator P. Note that 
P{x,y,z) = {x/z,y/z) is well defined and infinitely smooth only if z 7^ 0. 
When all points in the scene have depth bounded away from zero, then there is 
guaranteed a neighborhood of B around 6^ref wherein all points continue to have 
depth bounded away from zero. This ensures that the projection of all points is 
well defined, and hence, f{9) is infinitely smooth in this neighborhood. □ 

Note that, in contrast to affine articulations, where the corresponding OFM is 
smooth globally, the OFM associated with the pose manifold is smooth only over 
a neighborhood of the reference point. 

Next, we consider distances between optical flows and establish that the OFM 
is locally isometric. We begin by defining the following representation for rotation 
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matrices. Let uj = 
matrix defined as 



{ujx, ujy.uJz) ^ ]R . and let Vt^ G M^^^ be the skew- symmetric 







-^x 





UJy 





Noting that the matrix exponential e^^ is a rotation matrix, we define our articu- 
lations as 6^ = (cj, t) G M^, with t = {tx, ty, t^). 

Before we state a formal result, we need to introduce two assumptions that 
will help in simplifying the derivation. First, we assume that the imaging model 
is well-approximated by a weak perspective model [30]. In the weak perspective 
model, it is assumed that the variations in the scene depth are significantly smaller 
than the average depth of the scene. In such a case, the projection map P(x) = 
{x/z,y/z) can be well-approximated by 

-Pwp(^) (^/ ^av5 y/ ^av)? 

where is the average scene depth. As a consequence, translations along the z- 
axis are unobservable BOll : hence, we restrict the articulation space to the rotation 
of the camera and translation along x and y axes alone. Second, we assume that 
the rotation undergone by the camera is small. This enables us to approximate the 
matrix-exponential e^^ I + f]^. As a consequence of this assumption, we only 
obtain a local isometry result. With these two assumptions, we are ready to state 
a result on the distances between optical flows. 

Lemma 4. Consider the pose manifold under the assumption of the weak per- 
spective imaging model. Then the OFM is locally isometric. 

Proof. Without loss of generality, we denote our reference articulation as 6^ref = 0, 
which corresponds to identity rotation matrix and null translation vector. We are 
interested in the distance between two optical flow fields f{6i) and /(^2). Under 
our assumptions of weak perspective imaging model and small rotation, we can 
compute the distance between the optical flows to be 



/(^1)(X)-/(^2)(X) 



wp 

1 

^av 

1 

^av 



X 
1 



+ ti I — Pwp I Arefi?: 



X 
1 



(Aref - ^l,z) + ti,y) " (Aref (^2,x^ " ^2,z) + h.y) 

— Xrei{^l,x — ^2,x)y + Kef{^l,y — ^2,y) + {tl,x — h^x) 
Aref ~ ^2,x)x ~ Kei{^l,z ~ ^2,z) + ~ ^2,y) 
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Notice that both ti^z and t2^z are absent in the expression above. As noted above, 
this is due to the assumption of weak perspective imaging model which makes the 
translation of the camera along the z-axis unobservable. A key observation is that 
the articulation parameters 9i = (cji, ti) and 62 = {002, ^2) can be expressed as a 
function that is linear in the expression above. Hence, 

m)(x)-/(^2)(x)=^(x) 

Finally, summing over x G A', we obtain 

Hence, the OFM is locally isometric. □ 
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